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INVENTIO 



This invention relates>4o^^^oftware process which^ makes an existing two-dimensional 
image defocused in order to Greatesoiiincrease the sense Vf field depth. 

BACKGROUND DESCRIPTION OF RELATED. ART 



Producing a realistic two-dimensional Wage has been,^one^ the important fields of image 
processing. An image which is in good^foSijs^^l^raipQrt^^ part of reality but not enough. On 
a two dimensional image, good feelmg^oP^ing an ob^ec)^\vhich is originally a three 
dimensional object, often comes fi-om other blurred objects that ai^iout of focus giving a sense 
of field depth. In other word, if everydiing seen is in focus, the^im^ge would look unrealistic. 

v^li^n 



There are two cases when unrealistic ihiages can occur. One 



a computer generates 




images through three-dimensional rendering where there is no information on camera optics is 
available. Without camera opftcs, objects can not be defocused and realistic images can not be 
constructed. Another is when Camera optics has very deep focal depth often seen with recent 
digital cameras because their optics have very short focal length. All areas of an image 
produced by digital cameras are often in focus resulting in very poor sense of field-depth. 



Light ray tracing is the straight /approach to defocus images for computer generated 
three-dimensional objects. There hayb been several such works or inventions including U.S. 
Patent No. 5,986,659 to Gallery et il.\ U.S. Patent 6,028,606 to Kolb et al., and U.S. Patent 
4,731,864 to Modlsrnrti&se,,are based cJn a light-ray tracing process that simulates light rays 
emitted from aX^bject and passji^ through a camera lens, aperture (sometimes called stop or 
iris), shutter and mjr^. In su^k'a raysfa-aciifc process as well as in actual optics, if a very small 
spot such as a star bec5ffi^defocused>it juk looks like a disk over which the original amount 
of light is uniformly dis^bute. 4JTie sh^pe bf the disk is identical to the shape of a camera 
aperture. The original/^potis ^^Scused tAa (^i^k with its edge sharply defmed. This disk may 
be called a defocusea^isk. The size of the disk depends on the distance between an emitted 
point and the camera optics. l\the image^^ moreVocused, the size of the disk becomes smaller. 



A defocused image is summatkin^of^ere defocu^d disks form all points or pixels. Often the 
defocused disks from brid rt-^o^ntsj -emain seen^as^isks after the summation, giving a good 
sense of field depth, "^^se sii^ulations reproduce such defocused disks, although their 



ii^ulati 

processes are very detailbd and precise and therefc^re 
valuable to study or design a camera or an optic 
sense of field depth to human e) 



)st computing time very much. They are 
^tem but extravagant in giving a good 



For the images taken wit^i'l'^igitaTcani^^a-orLim^ges taken with deep focal depth, people 
sometimes try to use existitm software products to improveVhe sense of field depth. None of 
them, however, reproduce the^d^cused disks essential for sensing field depth as mentioned 
above. In these products such as^^oBe*SvsteinJiic.'s Photoshop or Jasc Inc.'s Paint Shop Pro, 
defocusing images is often misunders tood as Gaussian^blurring or other types of blurring. The 
Gaussian blurring distributes light a^cording^ "cl^lSSiaiudis^bul^on and does not produce the 
focused disks. 



To make matter worse, these products appIy^Bi^ 
not on the amount of light. This is a very 
stored in a pixel of an image corresponds 

sensitivity of human eyes, both of which ar\the amount of light nht \i linear scale but in 



^^CQcalopera^n on a pixel value itself, 
Iders^tiingVn this field. The value 
density of photq^aphic films or to the 





logarithmic scale. On the contrafy, actual physical process requires operation on the amount of 
light in linear scale. One photon plus one photon is two photons and one unit of light-energy 
plus one unit of light-energy is t^o units of light-energy. If the scale is in logarithmic scale, 
the result is indeed invalid. Since their defocusing-operation is applied directly to a pixel value, 
the results are invalidly and unrealfetically produced defocused images. The film or human 
eyes have a limited dynamic range for a large amount of light. These situations become most 
apparent when the characteristics of films or human eyes are correctly handled. If the amount 
of light is very large, saturation in sensiWity occurs and the light appears just white in films or 
eyes. Consider a datum in a pixel for briW light, for example, such as 253 or 254 in eight-bit 
range (0 to 255). The difference of just one digit in this brightest range may amount ten times 
or more difiFerenceJnJhe^Momit of light Energy or number of photons) in the real world. If a 
software procep does not account-^for^the ^al mount of light and create a defocused image, 
the effect is completely underestimatpd\resiming in a very unrealistic look. In such a case, a 
bright spot in an image spreads orJ/slightl^bnghter blur to surrounding pixels. A bright point 
in an image should sprea d^as-^ ^a^dMg defocused disk to surrounding pixels as actually seen 
in real photography. 



All the software proe^ses heretofore 
disadvantages: 



lown suffer from one(s) of the following 



(1) Light-ray tracing software costs computing time. 

(2) Existing image-processing^jDroducts do not prod{ic*e correct defocused images. They can 
not produce the defocused dis ks->^.\^ch argyt^ ^elsential part of defocused images as 
actually seen in the real world?^^ 



(3) Existing image-processing\products do not treat the amount of light correctly. The 
summation of amoimt of ligWshould be taken in linear ^ace not in logarithmic scale. 



BRIEF SUMMARY 



In accordance with the present invention, a software process\;omprises converting amount 
of light in logarithmic scale to linear scalejf neceslsary, spreadin^ight into a shape of aperture 
creating defocused disks, summation of the'^^l^^^^iskg.g'oii^V^l pixels, and converting 
amount of light back to logarithmic scale. 




Objects and Advant^e 
Several objects and advantages of the present invention are; 



(a) to provide a software process which can produce realistic defocused images without 
tracing light-rays to reduce computing time; 



/ 



3) to provide^^a^fbvare pipcess which can produce appealing defocused images by directly 
^^pr oducin g^ ^ks that are the most important part defocused images; and 

(c) to provide a software process which can produce realistic defocused images by treating 
amount^of'light'contectly in its process. 

f . r \\ 

Further objects and advantages are to provide a software process which allows to change the 
shape ofyaperturei^ (sometimes called stop or iris) to emulate actual apertures in existing 
cameras fors^ecia^efifects Wshe wants to create, to provide a software process which allows 
the effect of col5^•abe^^atipn by having color-dependent aperture information, and to provide a 
software process--,^iich allows toAmodify the transformation characteristic between 
"cel dafeftr'a 

wants. 





DRAWING FIGHRES 



In the drawingsTclosely related figures have the same number but different alphabetic suffixes. 



FIG 1 is a block\liagram of this invention. 

ion^^Orve 

amount of light which is in linear scale. 

. r — ^ \ 

FIGS. 3 A and 3B illustrate how a pixel^atum in an original image is defocused or spread over 
neighbor pixels in case oP^^entaeonal aperture. \ \ 



FIG 2 shows the transformatioalictov^^tween \ pixel datum which is in logarithmic and 



FIGS. 4 A to 4C show an example result of this so 
with one of existing software producfsf 




process and comparison to the results 
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DETAILED/DESCRIPTION 

FIG 1 - Block diagram ^^^^^^^^^^^ ^ 

FIG 1 shows a__^^k^di|gram of this\invention. In computer memory, the process stores 
three sets of input^data, an original color inriage 16, a shape of aperture 14, and an empty color 
image 12 to be used for a resulting image 2^^ The original color image 16 may be stored as a 
file on a compuW or transmitted from elspwhere to the computer. Then the image 16 is read 
into an area of compter memory and nmy be shown on a computer display. The image 16 
consists of data for ea^gxdo j^^d^ptlona llv may carry distance information for each pixel. 
The aperture shape^^^'^rnay also be a file on the computer or transmitted from elsewhere to the 
computer, then is read into an area of computen memory. The aperture data 1 4 could be a 



bitmap image of arbifeai:3^size with a number of pixels large enough for calculating defocused 
disks, or may be vector -^S^^^attu?£Q^ic representation alternatively used for calculating 
defocused disks. The empty image 12 is an area ofsmemory on the computer whose value is 
set to be zero, to which defocused=disks>are ^added o n\o make a final defocused image 28. 



A box 20, labeled as "ionvert pixel scale", represents a conversion of the scale of pixel data 
to linear scale correspondin^Ho^g^rcct amount of light, ^he scale of pixel data in the image 16 
is typically in logarithmic scale^^^^^^^'Saiieunt^f light or may be some similar scale to 
logarithmic. The pixel data may^e density recorded inVlk^tographic films or digitized data in 
a digital camera where the scaling is similar to that of phptpgraphic films. The relation of two 
scales, often called a characteristibxcurve or transformatic n\function, are shown in FIG. 2 and 




hi 



described in next section. 



A defocused image is created by spreading light-data over surrounding pixels according to 
the shape of a given aperture 14. This takes place in a box 22, labeled "calculate defocused 
disk". The total amount /)f light in 22 is conserved but equally distributed to an area 
determine'S^by aTgiVen aperture shape 14. 




A box 24, "add defocused disk" denotes that each defocused disk, which is produced from 
each original pixel, is co-added to a resulting image 12. 



^ 

After defocused disks from! the all pixels are co-added, the resulting image is complete but 
the scale of pixel value is in flinear scale (amount of light) which is not suitable for viewing 
^ with eyes. A^box 2^ "restore' pixll scale", coverts the linear pixel scale to original logarithmic 
2 scale inversely usjn ^die ^aracteristic curve described in FIG 2. The final image 28 may be 
''■4 displayed on a computer display by^ transferring the data to image memory, stored as a data file 
on an external'storagS^r directly transferred to other locations over network. 



FIG 2 - Characteristics curve 




Typical relations3etwe.^,aj^^l^atum and amount of light are shown as curves in FIG 2, 
often called characteristics curves. Pixel datum is plotted against amount of light for some 
^ typical relations^n FIG 2. The value of a pixel datum is expressed with eight bits ranging from 
^ 0 to 255 for eachfcolo^^hich is very typica^n a computer. The axis for amount of light is in 
logarithmic scale. The imit*o^:;aniaunt^of light here is arbitrary. It is a multiple of physical 
energy or number of photons. 




The thick solid curve 30 in the figure is one of the ch8iracteristics curves of actual 
photographic films prbv4ded by Fuji Photo Film^Inc. In this case, the pixel datum represents 
the image density on the filnr"*Hie..md dle part of t^e curve 30 is nearly straight indicating the 
pixel datum is approximately in logaridmiirscate insierms of amount of light. The curve 30 in 
detail, however, does not fit within the range of eighfcbits as shovm in the figure, where the 
curve 30 goes slightly below 0 or above 255. The important feature in the film characteristics 
is its behavior for very small\^ount of light and th^for very large amount of light. The 
density does not become exactly zero easily for low ligHt level as seen in lower left part of the 
curve running nearly horizontally tothe^Ig ft^ At hi^ ^^|jght\evel, the density does not become 
completely opaque easily as seen in upper right part of the^curve running nearly horizontally 




to the right. The later behavior is very important to create a realistic image in out of focus. 
When an image is in focus, a bright spot in the image occupies very small area, but in terms of 
amount of light, it can haveVlO to 100 times more than surrounding pixels. When the image 
becomes defocused, this largAamount of light on this spot spreads over the surrounding pixel 
resulting in significant effectsAThis .is one of the reasons why existing software products are 
incapable of creating realistic defocused images. They do not convert pixel data to amount of 
light; ther^foTe^e^effect of a brighf spot is completely under estimated. Even if the data come 
fi-om a dWtal camera, this^onv^sion is necessary because manufactures of digital cameras 
often use shmlar characteristi^u^es as photographic films. Other curves in the figure, thin 
solid line 39/^ott e^l Ij qe'^S, dqsyotted line 34 and two-dash dotted line 32, represent 
exaggerated examples where he/shW^ts to create more artistic effects than actual physical 
process. 




The most important^art defoeusedVmage visually is a defocused disk. After a pixel 
datum is converted to linearmnount of liglrt^ it is spread to surrounding pixels within the area 
whose shape is detOT]Qinfiil_m)m a shape %f a given aperture, with total amount of light 
conserved. Such ca^for a pentagon aperturem2 is illustrated in FIGS. 3A and 3B. The height 
of each column represents amoimt of light. InMal datum in a pixel is illustrated in FIGS. 3 A as 
a single column MO. Through our softwdfeX process, the amount of light is spread to 
surrounding pixels as^sliown in FIG. 3B wtfere^the shape of assumed aperture 42 is pentagonal. 
All of spread pixels have"tBe=^saBae:^e^t-4hat'is equal to the original height of 40 divided by 
the number of pixels in 44. We call a resulting spread area a defocused disk. The size of a 
defocused disk is determined by a user depending on his/her needs or by the distance 



ding on his 

information associated witha"'given pixel. AlthoughVlGS. 3A and 3B show illustrations for a 
single initial pixel, this process is applied'^-all pixels in an image. A defocused image is the 
summation of all of the defocused-disksJ 



FIGS. 4A-4C - Sample image and comparison 




Sample resuhs of this invention ar?§^^^hi_FIG 4BWth a comparison to one of existing 
software products in FIG. 4C. FIG .4^Ts an image before applying this software process. The 
image is printed in black and whitelfor application purpose but originally in color. FIG 4B is a 
resulting image after this software process with a pentagonXaperture, while FIG. 4C is the 
image created by "Gaussian blur" oiA^obe Systems Inc.'s U*hotoshop, which is the most 






popular product used for processing digital images. A striking difference is defocused disks 
seen in FIG 4B but not seen in FIG. 4C. The Gaussian blur simply blurs a pixel according to a 
Gaussian distribution which is not Vhat actually happens in real optics. Instead, a defocused 
image is produced by defocused disKs determined by an aperture which is sometimes called an 
iris or stop in optics. That is what is\een in FIG 4B. Another feature seen in FIG 4B is that 
these defocused disks are very brightWd sparkling. This is because this invention correctly 
computes the amount of light to be spread by converting pixel data to amount of light in linear 
scale. 



Advantage 

From the description abov^ 
images become evident: 




Lumber of Advantages of my software process for defocused 



(a) The direct use of {an aperture|shape forta defocused disk will reduce amount of computing 
time largely compare'd to existing lighy^ray ^^cing software processes. The reduction of 
computing time will penhit^casual us^^of^my sWware process among many end-users who 
own only limited computer resourc 




(b) The ability of changing aperture'^shapes enables Wistic visual efifects in defocused images 
depending on his/her artistic taste since my software process is not based on ray tracing of 
existing camera optics, user can change an/apertu^ shape to whatever shape he/she likes 
without increasing computingtiffiesignific^tly, which would be required in the case of ray 
tracing software process foc^om^exshape.,,.^^ 

(c) The use of a different apertute^^j^pe for each color can mimic a color aberration of optical 



lenses without increasing computing time as>tn;^ case of ray tracing. This can be very useftxl 
for creating an artistic effect appeali ng to hum an eyes. 



(d) The conversion of pixel datk in logaritimiic scale toXUnear scale before computing 
defocused disks can provide realisticS^jsual effects of bright sparkling defocused disks which 
are essential part of a defocused image but'trm^gletel^ missing iq the prior software processes. 



(d) The modifying the scale conversionnn the previous advanmge^will permit an effect which 
does not exist in real world but very usefiil for an artistic expression. 





Operation 

This software process first allocates memory for three types of images, an input image 1 6 to 
be defocused\ aperture information 14 and an output image 12. The allocation can be done in 
conventional way on a computer. The input image data 16 may reside on a data-storage device 
such as a hard disk or data tape, or elsewhere on network. The input image data can be black 
& white data or color data. The input image data may be a result of scanning a photographic 
film, digital data already stored on a digital camera, or a result of software image generation. 
TKe'3ataris-then rea^ into the memory for ftirther processing by the computer. The contents of 
m«nory allocat^ foAthe output image 12 are all set to be zero at the beginning. The aperture 
infomatijon^P^can^be^ither a bitmap image similar to the input image 14 or a mathematical 
formula representing\s\hape. The aperture information 14 can be different for each color if a 
user wants^t^reate an\fFect similar to the color aberration. 




G At "loop over all pixels" %^ in FIG. 1 , the process initiates a loop operation over the pixels 

^ of the input image. In the loop, each pixel is processed and a defocused disk is added to the 

in output im'age^til all the pixek are processed. The scale of datum in each pixel of the input 

UJ image is convertec^li?!^ scale. ak 20. The scale of input pixel data is often in logarithmic scale 

'j' or a scale^vety^m^ to that.Vhe conversion shown in 20 is always necessary for scanned 

3 data and Aata from a digital camera-. If the pixel data come from a computer-generated image 

such as a resulj^from three-dimei^icM^l rendering, the scale of the pixel data may be linear in 
£ terms of amount of>J[ig ht. In this case, the conversion shown in 20 may be skipped. The 

p conversion is d5nr3ng^^"diaraetenfetics curve whose examples are shown in FIG. 2. The 

^ nominal choicLof the characteristic cui\e is the curve 30 for photographic film but may be 

altered or changed^'deijending o^n his/her arristic taste. 

After the scale conveig ion.2Dlthe ^ta is uniformly distributed to a defocused disk at a box 
22 as illustrated in FIOsT^A ancl^Bpfhe"shape of a defocused disk is determined from the 
aperture informationlH. The sizliof a defocused disk is determined by a user or it can be 
computed from distanccvMbrmation attached toVthe pixel. Based on this shape and size, the 
pixel addresses of a defocuse3^iSlfeiTU^g^^^ut\mage 12 are determined. Each pixel of a 
defocused disk has the value^equaPtonhe converted^linear value of an input pixel divided by 
the number of pixels that a defocused disk occupies-^^ach pixel value of a focused disk is 
added to a corresponding alidress of the output image^^£. Thus a defocused disk is calculated 
for each input pixel and addeH^to a memory area of the optput image. 



m 
m 

m 



The process then goes to\a next input pixel, converts it to linear scale, calculates a 
defocused disk, and adds the disk data to the output image. It repeats this loop until all input 
pixels are processed. At this point, the resulting defocused image still has all of data in linear 
scale. At "restore pixel scale" 26i the scale of the pixel data in the output image is converted 
back to logarithmic-like scale using the same characteristic curve at 20 in the reverse way. The 



final defocused^image may,^ 
data storage, or transferred to 01 



di^ayed on a computer screen, saved as an image file on a 
^cation over network. 




Conclusion, Rariiifi^tion§i^*dnd Scope 



Accordingly, the reader will see thit tUe software process of this invention can be used to 
create realistic defocused iinages^vvithou^ using a large amount of computing time. A 
defocused image^created by this process correctly has defocused disks actually seen in real 
photograph. These disks are 'beautifully seemfor a bright spot creating a realistic look of a 
defocused image and giving a^good sense of fie^d depth. The reader will see that this software 
process is usefiil to increase Jrr^^re the field depth of images taken with recent digital 
cameras with a affordable computi^ resources 

Furthermore, the software process has%e additlpnal^^dvantage in that 
• it permits control oh the aperture shape and size; 



the 



it permits modification or atterin 
it permits emulation of color aberrafioiT; 




racteristic curve; and 



Although the description above ^ 



0 

Thus, the scope of the invention should be det& 
legal equivalents, rather than by the example-given 




)ntains\ many specifications, these should not be 
constructed as limiting the scope of the invention but^as merely^providing illustrations of some 
of the presently preferred embodiments of this invention. 





ined by the\appended claims and their 




